Writing Style Transfer¶

This notebook shows our writing style transfer network based on cycle GAN. The network design to transfer positive sentences into negative ones and the opposite.

import torch
import csv
import nltk
import os

import pandas as pd
import plotly.offline as py
import plotly.graph_objs as go
import numpy as np
from nltk.corpus import stopwords

from tqdm import tqdm
from nltk.translate.bleu_score import sentence_bleu, SmoothingFunction

py.init_notebook_mode(connected=False)

nltk.download('stopwords')

[nltk_data] Downloading package stopwords to /root/nltk_data...
[nltk_data]   Unzipping corpora/stopwords.zip.

True

def configure_plotly_browser_state():
  import IPython
  display(IPython.core.display.HTML('''
        <script src="/static/components/requirejs/require.js"></script>
        <script>
          requirejs.config({
            paths: {
              base: '/static/base',
              plotly: 'https://cdn.plot.ly/plotly-latest.min.js?noext', 
            },
          });
        </script>
        '''))

Get the repo files

!git clone https://github.com/eliorav/writing-style-transfer.git

Cloning into 'writing-style-transfer'...
remote: Enumerating objects: 45, done.
remote: Counting objects: 100% (45/45), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 45 (delta 4), reused 45 (delta 4), pack-reused 0
Unpacking objects: 100% (45/45), done.
Checking out files: 100% (32/32), done.

os.chdir('writing-style-transfer')

import dependencies

from GAN.CycleGan import get_cycle_gan_network, get_criterions, get_optimizers, get_schedulers
from Utils.DatasetLoader import load_dataset

batch_size = 16

Define the device

cuda = torch.cuda.is_available()
device = torch.device('cuda' if cuda else 'cpu')
print("device:",device.type)

device: cuda

Load the Dataset

source, iterators = load_dataset(batch_size, device)
train_iterator, validation_iterator, test_iterator = iterators

Explore the Data¶

from constants import DATA_FOLDER, POSITIVE_FILE_EXTENSION, NEGATIVE_FILE_EXTENSION, DATASET_TYPES, TRAIN_TYPE, TEST_TYPE, VALIDATION_TYPE

def average_words(x):
  words = x.split()
  return sum(len(word) for word in words) / len(words)

stop_words = stopwords.words('english')

for dataset_type in DATASET_TYPES.keys():
  for extension in [POSITIVE_FILE_EXTENSION, NEGATIVE_FILE_EXTENSION]:
    dataset=pd.read_csv(f"{DATA_FOLDER}/{DATASET_TYPES[dataset_type]}{extension}", delimiter="\t", names=["sentence"])
    dataset["word_count"] = dataset["sentence"].apply(lambda x: len(x.split()))
    dataset["char_count"] = dataset["sentence"].apply(lambda x: np.sum([len(ch) for ch in x.split()]))
    dataset["average_word_length"] = dataset["sentence"].apply(average_words)
    dataset["stopword_count"] = dataset["sentence"].apply(lambda x: len([word for word in x.split() if word.lower() in stop_words]))
    dataset["stopword_rate"] = dataset["stopword_count"] / dataset["word_count"]

    dataset_by_word_count = pd.DataFrame(dataset.groupby(['word_count']).count()).reset_index()
    
    print(f"Display sentences info and statistics for dataset type: {DATASET_TYPES[dataset_type]}{extension}\n\n")
    print(f"dataset size: {dataset.shape[0]}\n")

    display(dataset.head(10))
    
    print("\n\nsentences statistics")
    display(dataset.describe())

    print("\n\nnumber of sentences per word count")
    display(dataset_by_word_count[['word_count','sentence']])

Display sentences info and statistics for dataset type: train.pos


dataset size: 99454


sentences statistics


number of sentences per word count

Display sentences info and statistics for dataset type: train.neg


dataset size: 99039


sentences statistics


number of sentences per word count

Display sentences info and statistics for dataset type: test.pos


dataset size: 20000


sentences statistics


number of sentences per word count

Display sentences info and statistics for dataset type: test.neg


dataset size: 19787

Prepare the networks¶

G_INPUT_DIM = len(source.vocab)
G_OUTPUT_DIM = len(source.vocab)
SOS_IDX = source.vocab.stoi['<sos>']
PAD_IDX = source.vocab.stoi['<pad>']

Initialize the generators and discriminators

g_ab, g_ba, d_a, d_b = get_cycle_gan_network(G_INPUT_DIM, G_OUTPUT_DIM, device, PAD_IDX, SOS_IDX, True, True)

Display the models¶

Generator network
The network in a sequence to sequence network based on Sequence to Sequence Learning with Neural Networks

print(g_ab)

Seq2Seq(
  (encoder): Encoder(
    (embedding): Embedding(9887, 256)
    (rnn): LSTM(256, 512, num_layers=2, dropout=0.5)
    (dropout): Dropout(p=0.5)
  )
  (decoder): Decoder(
    (embedding): Embedding(9887, 256)
    (rnn): LSTM(256, 512, num_layers=2, dropout=0.5)
    (out): Linear(in_features=512, out_features=9887, bias=True)
    (dropout): Dropout(p=0.5)
  )
)

Disdiscriminator network
The network in a sentiment classifier based on Bag of Tricks for Efficient Text Classification

print(d_a)

Discriminator(
  (embedding): Embedding(9887, 256, padding_idx=1)
  (fc): Linear(in_features=256, out_features=1, bias=True)
)

Test the Generators¶

from GAN.train import evaluate_cycle_gan

Losses

criterion_g_ab, criterion_g_ba, criterion_gan, criterion_discriminator, criterion_cycle, criterion_identity = get_criterions(
    PAD_IDX, device)

loss_gan_ab, loss_gan_ba, bleu_score_a, bleu_score_b = evaluate_cycle_gan(
    source,
    device,
    g_ab,
    g_ba,
    d_a,
    d_b,
    iterators[2], # Test data
    criterion_gan
)
print(f'\nloss_gan_ab: {loss_gan_ab} | loss_gan_ba: {loss_gan_ba} | bleu_score_a: {bleu_score_a} | bleu_score_b: {bleu_score_b}')

100%|██████████| 1250/1250 [01:47<00:00,  8.17it/s]

loss_gan_ab: 2.3098510100841523 | loss_gan_ba: 1.4806003584086895 | bleu_score_a: 37.46300759913652 | bleu_score_b: 40.092413417641254

Display results¶

from Utils.Services import get_sentence_from_tensor

res_ab = []
res_ba = []

d_a.eval()
d_b.eval()
smoother = SmoothingFunction()

with torch.no_grad():
  with tqdm(total=len(test_iterator)) as pbar:
    for i, batch in enumerate(test_iterator):
      pbar.update(1)

      # Set model input
      real_a = batch.src.to(device)
      real_b = batch.trg.to(device)

      _, fake_b = g_ab(real_a, 0)
      _, fake_a = g_ba(real_b, 0)

      # Save A to B scores     
      real_a_sentences = get_sentence_from_tensor(source, real_a)
      fake_b_sentences = get_sentence_from_tensor(source, fake_b)
      for real_a_sentence, fake_b_sentence in zip(real_a_sentences, fake_b_sentences):
        bleu_score = sentence_bleu([real_a_sentence], fake_b_sentence, smoothing_function=smoother.method4) * 100
        res_ab.append({"Original": " ".join(real_a_sentence), "Transformed": " ".join(fake_b_sentence), "Bleu Score": bleu_score})

      # Save B to A scores   
      real_b_sentences = get_sentence_from_tensor(source, real_b)
      fake_a_sentences = get_sentence_from_tensor(source, fake_a)
      for real_b_sentence, fake_a_sentence in zip(real_b_sentences, fake_a_sentences):
        bleu_score = sentence_bleu([real_b_sentence], fake_a_sentence, smoothing_function=smoother.method4) * 100
        res_ba.append({"Original": " ".join(real_b_sentence), "Transformed": " ".join(fake_a_sentence), "Bleu Score": bleu_score})

100%|██████████| 1250/1250 [01:37<00:00,  8.33it/s]

Generator positive to negative results¶

df_ab = pd.DataFrame(res_ab)
df_ab["Original Length"] = df_ab["Original"].apply(lambda x: len(x.split()))
df_ab_bleu_by_length = pd.DataFrame(df_ab.groupby(['Original Length'])['Bleu Score'].mean())
df_ab_bleu_by_length = df_ab_bleu_by_length.reset_index()
df_ab_bleu_by_length["network_type"] = "G_AB"
df_ab.head(20)

df_ab[(df_ab['Bleu Score'] < 40) & (df_ab['Original Length'] < 8) & (~df_ab["Original"].str.contains('<unk>')) & (~df_ab["Transformed"].str.contains('<unk>'))].sort_values(by=['Bleu Score'], ascending=False).head(20)

Generator negative to positive results¶

df_ba = pd.DataFrame(res_ba)
df_ba["Original Length"] = df_ba["Original"].apply(lambda x: len(x.split()))
df_ba_bleu_by_length = pd.DataFrame(df_ba.groupby(['Original Length'])['Bleu Score'].mean())
df_ba_bleu_by_length = df_ba_bleu_by_length.reset_index()
df_ba_bleu_by_length["network_type"] = "G_BA"
df_ba.head(20)

df_ba[(df_ba['Bleu Score'] < 40) & (df_ba['Original Length'] < 8) & (~df_ba["Original"].str.contains('<unk>')) & (~df_ba["Transformed"].str.contains('<unk>'))].sort_values(by=['Bleu Score'], ascending=False).head(20)

Bleu Score¶

df_ab_display = df_ab_bleu_by_length[(df_ab_bleu_by_length['Original Length'] <= 15) & (df_ab_bleu_by_length['Original Length'] >= 2)]
df_ba_display = df_ba_bleu_by_length[(df_ba_bleu_by_length['Original Length'] <= 15) & (df_ba_bleu_by_length['Original Length'] >= 2)]

configure_plotly_browser_state()
ab_bat = go.Bar(
    x=df_ab_display["Original Length"],
    y=df_ab_display["Bleu Score"],
    name = 'G_AB',
)

ba_bar = go.Bar(
    x=df_ba_display["Original Length"],
    y=df_ba_display["Bleu Score"],
    name = 'G_BA',
)

data = [ab_bat, ba_bar]

layout = go.Layout(title = 'Bleu score per sentence length',
                   xaxis = {'title': 'Sentence Length'},
                   yaxis = {'title': 'Bleu Score'}
                   )

fig = go.Figure(data=data, layout=layout)

py.iplot(fig)

G_AB bleu score per sentence length¶

df_ab_display[['Original Length','Bleu Score']]

G_BA bleu score per sentence length¶

df_ba_display[['Original Length','Bleu Score']]

	sentence	word_count	char_count	average_word_length	stopword_count	stopword_rate
0	went in for a lunch.	5	16	3.200000	3	0.600000
1	i know i'm going on about the salad ...	9	31	3.444444	4	0.444444
2	but it was perfect.	4	16	4.000000	3	0.750000
3	drink prices were pretty good.	5	26	5.200000	1	0.200000
4	the server, dawn, was friendly and accommodating.	7	43	6.142857	3	0.428571
5	very happy with her.	4	17	4.250000	2	0.500000
6	in summation, a great pub experience.	6	32	5.333333	2	0.333333
7	would go again!	3	13	4.333333	0	0.000000
8	this place never disappoints.	4	26	6.500000	1	0.250000
9	they even have great salads and grilled chicken.	8	41	5.125000	3	0.375000

	word_count	char_count	average_word_length	stopword_count	stopword_rate
count	99454.000000	99454.000000	99454.000000	99454.000000	99454.000000
mean	8.384238	38.490297	4.690603	3.645153	0.415087
std	7.797520	35.822840	0.934113	3.674323	0.156648
min	1.000000	2.000000	1.000000	0.000000	0.000000
25%	6.000000	26.000000	4.000000	2.000000	0.333333
50%	8.000000	37.000000	4.571429	3.000000	0.428571
75%	11.000000	49.000000	5.200000	5.000000	0.500000
max	1418.000000	6575.000000	25.166667	616.000000	0.900000

	word_count	sentence
0	1	1
1	2	17
2	3	6243
3	4	9063
4	5	9431
5	6	9412
6	7	9000
7	8	8807
8	9	8961
9	10	8671
10	11	8294
11	12	7917
12	13	7182
13	14	6446
14	20	1
15	21	1
16	22	1
17	67	1
18	170	1
19	462	1
20	1173	1
21	1187	1
22	1418	1

	sentence	word_count	char_count	average_word_length	stopword_count	stopword_rate
0	this place has gone down hill.	6	25	4.166667	3	0.500000
1	i've been going for years and the food quality...	13	54	4.153846	6	0.461538
2	the sauce selection is limited and not very fr...	12	65	5.416667	7	0.583333
3	love this place downtown but the scottsdale lo...	11	58	5.272727	5	0.454545
4	sat at bar for 10 min while bartender ignored us.	10	40	4.000000	3	0.300000
5	no menu, no water.	4	15	3.750000	2	0.500000
6	we walked out and they could have cared less.	9	37	4.111111	5	0.555556
7	main: caprese saladthe salad was okay - a bit ...	13	54	4.153846	4	0.307692
8	they keep there appointments on time and are p...	10	53	5.300000	5	0.500000
9	when he told me the price of the two tires it ...	14	49	3.500000	8	0.571429

	word_count	char_count	average_word_length	stopword_count	stopword_rate
count	99039.000000	99039.000000	99039.000000	99039.000000	99039.000000
mean	8.593271	38.417785	4.567089	3.898121	0.436306
std	11.608588	52.019543	0.974911	5.441465	0.157340
min	1.000000	6.000000	1.500000	0.000000	0.000000
25%	6.000000	26.000000	4.000000	2.000000	0.333333
50%	9.000000	37.000000	4.428571	4.000000	0.454545
75%	11.000000	49.000000	5.000000	5.000000	0.545455
max	2827.000000	12688.000000	70.500000	1268.000000	1.000000

	word_count	sentence
0	1	1
1	2	15
2	3	6494
3	4	7856
4	5	8395
5	6	8827
6	7	8939
7	8	8989
8	9	8993
9	10	8996
10	11	8605
11	12	8117
12	13	7671
13	14	7125
14	15	1
15	16	1
16	30	1
17	36	1
18	65	1
19	164	1
20	171	1
21	204	1
22	296	1
23	337	1
24	495	1
25	522	1
26	533	1
27	1255	1
28	1315	1
29	2827	1

	sentence	word_count	char_count	average_word_length	stopword_count	stopword_rate
0	phillipp is a great dentist, very friendly and...	9	52	5.777778	4	0.444444
1	i highly recommend this office for the nice sy...	13	61	4.692308	5	0.384615
2	plumbsmart provided superior service from begi...	8	52	6.500000	2	0.250000
3	thank you to wayne and the whole team!	8	31	3.875000	4	0.500000
4	best wonton soup ever!!!	4	21	5.250000	0	0.000000
5	delicious food and great service.	5	29	5.800000	1	0.200000
6	definitely will be back!	4	21	5.250000	2	0.500000
7	had the kale grits and would definitely have t...	10	48	4.800000	5	0.500000
8	menu choices are unique, farm fresh and delici...	8	43	5.375000	2	0.250000
9	these burgers are phenomenal!	4	26	6.500000	2	0.500000

	word_count	char_count	average_word_length	stopword_count	stopword_rate
count	20000.000000	20000.000000	20000.000000	20000.000000	20000.000000
mean	8.334800	38.261600	4.688364	3.634250	0.415825
std	3.312123	15.179841	0.938014	2.089209	0.157724
min	1.000000	1.000000	1.000000	0.000000	0.000000
25%	5.000000	26.000000	4.000000	2.000000	0.333333
50%	8.000000	37.000000	4.571429	3.000000	0.428571
75%	11.000000	50.000000	5.200000	5.000000	0.500000
max	14.000000	123.000000	18.250000	11.000000	0.875000

	word_count	sentence
0	1	1
1	2	6
2	3	1305
3	4	1794
4	5	1918
5	6	1853
6	7	1882
7	8	1751
8	9	1726
9	10	1766
10	11	1615
11	12	1590
12	13	1439
13	14	1354

	sentence	word_count	char_count	average_word_length	stopword_count	stopword_rate
0	total bill for this horrible service?	6	32	5.333333	2	0.333333
1	i checked online the pills can be had for 19 c...	12	45	3.750000	6	0.500000
2	avoid hospital ers at all costs.	6	27	4.500000	2	0.333333
3	met a friend for dinner there tonight.	7	32	4.571429	3	0.428571
4	the server only brought us one menu.	7	30	4.285714	2	0.285714
5	we received the starters and the mains at the ...	11	46	4.181818	7	0.636364
6	the wings, brisket, and mac and cheese were al...	11	50	4.545455	5	0.454545
7	food was somewhere between okay and decent.	7	37	5.285714	3	0.428571
8	service was somewhere between piss poor and te...	8	46	5.750000	3	0.375000
9	the food wasn't very good, way too greasy.	8	35	4.375000	4	0.500000

	word_count	char_count	average_word_length	stopword_count	stopword_rate
count	19787.000000	19787.000000	19787.000000	19787.000000	19787.000000
mean	8.625714	38.534442	4.563503	3.917421	0.435828
std	10.245515	46.386365	0.937895	5.070039	0.157860
min	2.000000	5.000000	1.666667	0.000000	0.000000
25%	6.000000	26.000000	4.000000	2.000000	0.333333
50%	8.000000	38.000000	4.428571	4.000000	0.454545
75%	11.000000	49.000000	5.000000	5.000000	0.545455
max	1318.000000	5959.000000	19.000000	624.000000	0.888889

	word_count	sentence
0	2	1
1	3	1303
2	4	1548
3	5	1645
4	6	1764
5	7	1806
6	8	1831
7	9	1743
8	10	1707
9	11	1764
10	12	1664
11	13	1565
12	14	1441
13	15	1
14	16	1
15	100	1
16	378	1
17	1318	1

	sentence	word_count	char_count	average_word_length	stopword_count	stopword_rate
0	our family loves the food here.	6	26	4.333333	2	0.333333
1	quick, friendly, delicious, and a great restau...	11	57	5.181818	3	0.272727
2	great food, great service.	4	23	5.750000	0	0.000000
3	their chicken is great, doesn't seem too fatty...	14	69	4.928571	7	0.500000
4	great food, drinks, and service, but the atmos...	12	59	4.916667	5	0.416667
5	definitely a must visit for locals and visitor...	9	46	5.111111	3	0.333333
6	you will not feel like you're in "pittsburgh".	8	39	4.875000	5	0.625000
7	quoted me a good price for the repairs as well.	10	38	3.800000	5	0.500000
8	flat out fantastic!	3	17	5.666667	1	0.333333
9	have been here twice and the croissants are th...	10	44	4.400000	7	0.700000

	word_count	sentence
0	2	7
1	3	1252
2	4	1837
3	5	1805
4	6	1876
5	7	1778
6	8	1821
7	9	1841
8	10	1790
9	11	1667
10	12	1555
11	13	1376
12	14	1395

	sentence	word_count	char_count	average_word_length	stopword_count	stopword_rate
0	don't even think they realized we walked in.	8	37	4.625000	3	0.375000
1	however everyone at the bar noticed we walked ...	9	43	4.777778	3	0.333333
2	service was non existent at best.	6	28	4.666667	2	0.333333
3	not a good way for a new business to start out.	11	37	3.363636	5	0.454545
4	staff is not nice, very rude.	6	24	4.000000	3	0.500000
5	and it was a big mistake.	6	20	3.333333	4	0.666667
6	i explained what i wanted to the stylist, and ...	13	57	4.384615	6	0.461538
7	i was mortified and now trying to grow this hi...	12	49	4.083333	6	0.500000
8	nothing is blended and she butchered my hair.	8	38	4.750000	4	0.500000
9	and to make matters worse, she wasn't even fri...	9	44	4.888889	4	0.444444

	word_count	sentence
0	1	1
1	2	4
2	3	1317
3	4	1561
4	5	1717
5	6	1830
6	7	1731
7	8	1770
8	9	1836
9	10	1707
10	11	1716
11	12	1591
12	13	1538
13	14	1517
14	36	1
15	213	1
16	412	1
17	740	1

	Bleu Score	Original	Transformed	Original Length
0	18.665471	would go again	never again !	3
1	28.650724	great place !	horrible place !	3
2	18.665471	amazing show .	received book .	3
3	28.650724	go to swirl	go to ...	3
4	18.665471	10/10 would recommend	highly would ?	3
5	18.665471	thanks again guys	wait guys ..	3
6	28.650724	excellent food !	terrible food !	3
7	22.197111	good people .	good lines .	3
8	0.000000	visited july 2013	booked brand .	3
9	18.665471	amazing service !	terrible music !	3
10	22.197111	would definitely recommend	would recommend !	3
11	0.000000	wonderful staff .	terrible ... ...	3
12	18.665471	<unk> md	<unk> continually carla	2
13	18.665471	thank you jesus	if you ?	3
14	18.665471	pam <unk> florida	<unk> <unk> ?	3
15	18.665471	congratulations mia !	sorry ! !	3
16	30.408240	staff is really friendly and helpful .	staff is really helpful helpful either .	7
17	26.207968	everything was hot and tasted cooked !	food was cold and burnt off !	7
18	28.758338	just used them again july 2014 .	just used them early afternoon . .	7
19	26.207968	very reliable , professional and efficient .	very condescending , & disappointed .	7

	Bleu Score	Original	Transformed	Original Length
4265	38.44001	we were not disappointed !	we were not ! !	5
2649	38.44001	will definitely be returning .	will not be returning .	5
517	38.44001	i 'm a fan .	i 'm a expert .	5
2621	38.44001	the service is amazing .	the service is terrible .	5
2624	38.44001	i love this show !	i hate this show !	5
2629	38.44001	and it was amazing .	and it was wrong .	5
2630	38.44001	this hotel is gigantic .	this hotel is closed .	5
2641	38.44001	the service is wonderful .	the service is terrible .	5
2643	38.44001	this place is awesome .	this place is awful .	5
2648	38.44001	this place is amazing .	this place is terrible .	5
2656	38.44001	the food was outstanding .	the food was eh .	5
17683	38.44001	will always come back .	will never come back .	5
2682	38.44001	it was very delicious .	it was very bland .	5
2709	38.44001	this place is awesome .	this place is awful .	5
2716	38.44001	the decor is perfect !	the decor is dirty !	5
2722	38.44001	the food was amazing !	the food was terrible !	5
2723	38.44001	everyone here is fabulous .	everyone here is unacceptable .	5
2734	38.44001	there is no waiting .	there is no answer .	5
2735	38.44001	the service is great too	the service is too ?	5
4553	38.44001	seriously , go now .	seriously , go tonight .	5

	Bleu Score	Original	Transformed	Original Length
0	23.681572	the return flight was even worse .	the long crew are simply amazing .	7
1	43.472087	this place is a total joke .	this place is a hidden gem .	7
2	24.733286	save your time and your money .	bring your time with your request .	7
3	30.408240	this food was disgusting and bland .	this food was delicious and delicious .	7
4	26.259740	basically a repeat of others .	actually a variety of mine .	6
5	100.000000	awesome vibe and awesome service !	awesome vibe and awesome service !	6
6	38.330765	we were being skipped over !	we were being hungry over !	6
7	50.813275	beyond livid with this service !	pleasantly surprised with this service !	6
8	24.503243	what a fucking rip off .	what a fantastic hidden gem .	6
9	38.330765	always had a good experience .	always had a bad experience .	6
10	53.728497	i only have one phone .	i only have one issues .	6
11	100.000000	they do n't even <unk> .	they do n't even <unk> .	6
12	32.756476	wasting your time and lying .	enjoyed your time and jacuzzi .	6
13	66.874030	never will be back .	definitely will be back .	5
14	38.440010	wo n't be returning .	wo n't be happier .	5
15	28.662276	service was terrible .	service was excellent .	4
16	28.650724	never come back	will come back	3
17	18.665471	no call back	great coming back	3
18	28.650724	avoid this place	love this place	3
19	28.650724	worst service .	best service .	3

	word_count	char_count	average_word_length	stopword_count	stopword_rate
count	19840.000000	19840.000000	19840.000000	19840.000000	19840.000000
mean	8.580696	38.373538	4.577013	3.889919	0.434910
std	6.957646	31.237367	1.568913	3.598699	0.157548
min	1.000000	6.000000	2.000000	0.000000	0.000000
25%	6.000000	26.000000	4.000000	2.000000	0.333333
50%	8.000000	37.000000	4.428571	4.000000	0.454545
75%	11.000000	49.000000	5.000000	5.000000	0.545455
max	740.000000	3332.000000	183.000000	343.000000	1.000000

	Bleu Score	Original	Transformed	Original Length
859	38.44001	it came out clear .	it came out scottsdale .	5
306	38.44001	the server was forgetful .	the server was funny .	5
314	38.44001	would n't go back .	would n't go wrong .	5
313	38.44001	the food was bad .	the food was good .	5
7897	38.44001	so out of here !	so out of day !	5
311	38.44001	i made a reservation .	i made a mistake .	5
7902	38.44001	you guys are incompetent ..	you guys are seating ..	5
6166	38.44001	this place is ridiculous .	this place is amazing .	5
6169	38.44001	how does that happen .	how does that justice .	5
307	38.44001	our food was overcooked .	our food was delish .	5
305	38.44001	i was pretty disgusted .	i was pretty satisfied .	5
346	38.44001	bartenders are so rude .	bartenders are so friendly .	5
304	38.44001	i was so disappointed !	i was so pleased !	5
302	38.44001	wo n't be back .	wo n't be disappointed .	5
301	38.44001	this was in 2008 .	this was in heaven .	5
298	38.44001	service was so slow !	service was so fast !	5
296	38.44001	thank you , mrs .	thank you , roadrunner .	5
7907	38.44001	we were all disappointed .	we were all impressed .	5
294	38.44001	this place is gross .	this place is delicious .	5
7908	38.44001	service usually is good .	service seems is good .	5

	Original Length	Bleu Score
1	2	25.386273
2	3	22.472203
3	4	31.701177
4	5	29.408466
5	6	29.211100
6	7	28.883666
7	8	30.410867
8	9	34.977702
9	10	43.841166
10	11	44.353613
11	12	45.951316
12	13	44.594281
13	14	45.502815
14	15	43.295173

	Original Length	Bleu Score
0	3	20.564016
1	4	34.394561
2	5	36.870929
3	6	37.725837
4	7	34.098215
5	8	35.205391
6	9	37.938276
7	10	43.444882
8	11	45.571349
9	12	45.267087
10	13	45.004513
11	14	43.294112
12	15	41.959133